Skip to content

feat(native-eval): support OpenClaw candidate packages - #57

Open
vincentkoc wants to merge 3 commits into
mainfrom
fix/openclaw-candidate-artifacts
Open

feat(native-eval): support OpenClaw candidate packages#57
vincentkoc wants to merge 3 commits into
mainfrom
fix/openclaw-candidate-artifacts

Conversation

@vincentkoc

@vincentkoc vincentkoc commented Jul 29, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

What does this PR do?

Adds exact local OpenClaw package artifacts to native evaluation campaigns.

Why?

ShellBench needs reproducible pre-publication baseline/candidate runs without
publishing temporary npm versions or mutating warmed machines.

Fixes #56

Changes

  • add --openclaw-package-tarball campaign input and OpenClaw-only validation
  • stage, upload, verify, and install the exact package artifact
  • bind resume identity to package name, version, and SHA-256
  • record candidate provenance in run manifests and research audits
  • document the candidate workflow and failure modes

Exact proof

  • Candidate tarball SHA:
    933616f90607b43970d2aaaf1de051a5fe8a751e514882f69c557d56e15755d2
  • Exact c7a.8xlarge AWS run installed the package and exported three results.
  • Exported manifests preserved package name openclaw, version 2026.7.2,
    artifact filename, and SHA-256.
  • The run later exposed an independent Tool Search control bug tracked in fix(native-eval): make OpenClaw Tool Search controls explicit #61;
    that invalidates its score comparison, not the package staging proof.

Tests

  • 108 focused native-eval tests pass

  • Ruff passes on all touched Python files

  • fresh Codex autoreview reports no actionable findings

  • python -m pytest -q passes locally

  • python -m ruff check clawbench app.py scripts tests passes locally, or the change is docs-only

@clawsweeper clawsweeper Bot added rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. P2 Normal priority bug or improvement with limited blast radius. labels Jul 29, 2026
@clawsweeper

clawsweeper Bot commented Jul 29, 2026 •
edited
Loading

Copy link
Copy Markdown

Codex review: needs maintainer review before merge. Reviewed August 2, 2026, 3:15 PM ET / 19:15 UTC.

ClawSweeper review

What this changes

This PR adds an OpenClaw-only native-evaluation option that stages a local npm tarball, installs that exact artifact on leased hosts, and carries its identity through run manifests and research audits.

Merge readiness

⛔ Blocked until stronger real behavior proof is added - 4 items remain

Keep this PR open for maintainer review. The feature is not on current main and the implementation is coherent, but it introduces a trusted-local npm artifact execution path on leased hosts; its SHA pinning protects identity after selection, not artifact provenance. Likely related people: Vincent Koc is the original native-eval author and recent contributor for this area (high confidence).

Priority: P2
Reviewed head: 3da8ec53b7964b64469dafd3090abcf318d942c8
Owner decision: Required. See Decision needed.

Review scores

Measure Result What it means
Overall readiness 🧂 unranked krab (1/6) The implementation has a focused design and regression coverage, but it is not merge-ready because inspectable real-host proof is absent and the new execution boundary still needs maintainer acceptance.
Proof confidence 🧂 unranked krab (1/6) Needs stronger real behavior proof before merge: The PR body asserts a successful AWS candidate run, but the prepared proof contains only unrelated Blacksmith button images. Add redacted terminal output, runtime logs, or exported manifests that visibly show SHA verification, installed OpenClaw version, and recorded candidate provenance; after updating the PR body, request @clawsweeper re-review if a fresh review does not start automatically. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
Patch quality 🐚 platinum hermit (4/6) Security review found an item that needs attention.

Verification

Check Result Evidence
Real behavior Needs proof Needs stronger real behavior proof before merge: The PR body asserts a successful AWS candidate run, but the prepared proof contains only unrelated Blacksmith button images. Add redacted terminal output, runtime logs, or exported manifests that visibly show SHA verification, installed OpenClaw version, and recorded candidate provenance; after updating the PR body, request @clawsweeper re-review if a fresh review does not start automatically. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
Evidence reviewed 6 items Current main lacks this capability: The current main checkout has no local OpenClaw candidate-tarball flag or candidate-package provenance path; the proposed feature therefore remains necessary if maintainers accept its execution boundary.
Candidate identity and resume protection: The branch validates an OpenClaw-only package, compares its version with planned runs, stages it with mode 0600, stores its SHA-256, and rejects changed or missing candidate identity on resume.
Remote execution boundary: The branch copies the selected archive to each leased host, checks its SHA-256, and gives the archive to npm during bootstrap. That preserves selected-bit integrity but permits package lifecycle scripts from the supplied artifact to execute.
Findings None None.
Security Needs attention Accept the local package execution boundary: The supplied archive becomes an npm install input on the remote host, where lifecycle scripts may execute. Its SHA-256 validates byte identity after selection but does not establish a trusted producer.

How this fits together

ShellBench native evaluation packages a pinned runner and task suite, provisions remote hosts, bootstraps the selected harnesses, and exports run metadata for auditing. This change adds an OpenClaw candidate package to that controller-to-host bootstrap path and records its version and SHA-256 in campaign, toolchain, and run provenance.

flowchart LR
  A[Local OpenClaw tarball] --> B[Campaign controller]
  B --> C[Package metadata and SHA check]
  C --> D[Leased evaluation host]
  D --> E[Bootstrap npm installation]
  E --> F[Run and toolchain manifests]
  F --> G[Research audit export]
Loading

Decision needed

Question Recommendation
Should native evaluation campaigns permit a locally supplied OpenClaw npm tarball to execute during leased-host bootstrap when the operator provides the artifact and ShellBench records only its SHA-pinned identity? Accept trusted-artifact mode: Allow trusted campaign operators to use the local tarball input, with the documented SHA-pinned provenance boundary.

Why: Whether controlled campaign operators may introduce an arbitrary local npm artifact as a bootstrap source is a product security-boundary decision; source review cannot decide the acceptable trust and attestation policy.

Before merge

  • Resolve security concern: Accept the local package execution boundary - The supplied archive becomes an npm install input on the remote host, where lifecycle scripts may execute. Its SHA-256 validates byte identity after selection but does not establish a trusted producer.
  • Resolve merge risk (P1) - A local tarball accepted by this option can run npm lifecycle scripts during bootstrap on leased hosts; the SHA-256 check verifies the chosen bytes but does not attest who produced them.
  • Resolve merge risk (P1) - The PR body describes a successful AWS run, but no inspectable after-fix runtime output or exported manifest was available in the prepared proof artifacts, so reviewers cannot independently confirm the staged-host behavior.

Findings

  • [medium] Accept the local package execution boundary — scripts/native_eval/bootstrap_beast.sh:108
Agent review details

Security

Needs attention: The patch deliberately adds a trusted-local npm execution path; integrity checks are present, but maintainer approval is needed because package lifecycle scripts can execute on leased hosts without producer attestation.

Review metrics

Metric Value Why it matters
Patch scope 10 files affected; 823 added, 23 removed The change spans campaign staging, remote bootstrap, provenance exports, audit checks, documentation, and focused regression coverage.

Merge-risk options

Maintainer options:

  1. Accept the trusted-artifact boundary (recommended)
    Merge after a maintainer explicitly accepts that trusted local npm artifacts may execute their lifecycle scripts on leased evaluation hosts.
  2. Gate artifacts by provenance
    Add a concrete trusted-build signature or attestation check before local archives are accepted, then refresh real-host proof.
  3. Pause this capability
    Defer or close the PR if native evaluation should remain restricted to registry-installed OpenClaw packages.

Technical review

Best possible solution:

If maintainers approve trusted local artifacts for controlled campaigns, retain the OpenClaw-only scope, SHA-pinned resume identity, and audit provenance, document the operator trust boundary, and add redacted real-host proof showing verification, installation, and exported provenance.

Do we have a high-confidence way to reproduce the issue?

Not applicable: this PR proposes a new evaluation capability rather than reporting broken existing behavior. The stated AWS run has no inspectable runtime artifact in the prepared evidence, so it is not independently reproducible from this review.

Is this the best way to solve the issue?

Unclear: the source implementation is focused and preserves candidate identity well, but accepting a locally supplied package as executable bootstrap input requires maintainer security-policy approval before it can be called the best solution.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 884dd1bb5511.

Labels

Label changes:

  • add rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🧂 unranked krab and patch quality is 🐚 platinum hermit.
  • add status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs stronger real behavior proof before merge: The PR body asserts a successful AWS candidate run, but the prepared proof contains only unrelated Blacksmith button images. Add redacted terminal output, runtime logs, or exported manifests that visibly show SHA verification, installed OpenClaw version, and recorded candidate provenance; after updating the PR body, request @clawsweeper re-review if a fresh review does not start automatically. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
  • remove rating: 🐚 platinum hermit: Current PR rating is rating: 🧂 unranked krab, so this older rating label is no longer current.
  • remove status: 👀 ready for maintainer look: Current PR status label is status: 📣 needs proof.

Label justifications:

  • P2: This is a bounded native-evaluation feature with a meaningful security-boundary decision, but it is not an active product outage.
  • merge-risk: 🚨 security-boundary: Merging adds a locally supplied npm package as a remote bootstrap input, allowing its lifecycle scripts to execute on leased hosts.
  • rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🧂 unranked krab and patch quality is 🐚 platinum hermit.
  • status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs stronger real behavior proof before merge: The PR body asserts a successful AWS candidate run, but the prepared proof contains only unrelated Blacksmith button images. Add redacted terminal output, runtime logs, or exported manifests that visibly show SHA verification, installed OpenClaw version, and recorded candidate provenance; after updating the PR body, request @clawsweeper re-review if a fresh review does not start automatically. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Evidence

Security concerns:

  • [medium] Accept the local package execution boundary — scripts/native_eval/bootstrap_beast.sh:108
    The supplied archive becomes an npm install input on the remote host, where lifecycle scripts may execute. Its SHA-256 validates byte identity after selection but does not establish a trusted producer.
    Confidence: 0.98

What I checked:

  • Current main lacks this capability: The current main checkout has no local OpenClaw candidate-tarball flag or candidate-package provenance path; the proposed feature therefore remains necessary if maintainers accept its execution boundary. (scripts/native_eval/fleet.py:1, 884dd1bb5511)
  • Candidate identity and resume protection: The branch validates an OpenClaw-only package, compares its version with planned runs, stages it with mode 0600, stores its SHA-256, and rejects changed or missing candidate identity on resume. (scripts/native_eval/fleet.py:539, 3da8ec53b796)
  • Remote execution boundary: The branch copies the selected archive to each leased host, checks its SHA-256, and gives the archive to npm during bootstrap. That preserves selected-bit integrity but permits package lifecycle scripts from the supplied artifact to execute. (scripts/native_eval/bootstrap_beast.sh:97, 3da8ec53b796)
  • Feature-history ownership: The current native-evaluation controller and audit surface originate in Vincent Koc's native matrix-runner commit, with later OpenClaw trace work also authored by Vincent Koc; the proposed branch's three commits are by the same contributor. (scripts/native_eval/fleet.py:1, 69f75c6629c4)
  • Prepared proof inspection: The prepared media manifest contains two Blacksmith “Fix with Codesmith” button images only; neither shows the stated AWS candidate install, SHA validation, installed version, or exported provenance. (/home/runner/_work/clawsweeper/clawsweeper/artifacts/event/codex/proof-scratch/57/media-proof-manifest.json:1)
  • Applicable review policy: The fully read applicable AGENTS.md treats maintainer-authored items as non-closeable and requires current real-behavior proof for code-bearing changes before landing. (../AGENTS.md:47, 884dd1bb5511)

Likely related people:

  • Vincent Koc: Authored the native matrix-runner and recent OpenClaw trace work on current main, as well as all three commits in this candidate-package branch. (role: original native-eval feature author and recent area contributor; confidence: high; commits: 69f75c6629c4, 884dd1bb5511, 2cda5f84d5d4; files: scripts/native_eval/fleet.py, scripts/native_eval/bootstrap_beast.sh, scripts/native_eval/run_job.py)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

  • Add redacted real-host proof showing the candidate archive SHA verification, installed OpenClaw version, and resulting provenance fields.
  • Obtain an explicit maintainer decision on whether trusted local npm artifacts may execute during leased-host bootstrap.

Rating scale

Score Internal tier Crab rank Meaning
6/6 S 🦀 challenger crab Exceptional readiness
5/6 A 🦞 diamond lobster Very strong readiness
4/6 B 🐚 platinum hermit Good normal PR; ordinary maintainer review
3/6 C 🦐 gold shrimp Useful, but confidence is limited
2/6 D 🦪 silver shellfish Proof or implementation needs work
1/6 F 🧂 unranked krab Not merge-ready
N/A NA 🌊 off-meta tidepool Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (18 earlier review cycles; latest 8 shown)
  • reviewed 2026-08-01T00:25:47.301Z sha 3da8ec5 :: needs maintainer review before merge. :: none
  • reviewed 2026-08-01T06:14:50.813Z sha 3da8ec5 :: needs maintainer review before merge. :: none
  • reviewed 2026-08-01T10:56:42.938Z sha 3da8ec5 :: needs maintainer review before merge. :: none
  • reviewed 2026-08-01T12:21:53.837Z sha 3da8ec5 :: needs maintainer review before merge. :: none
  • reviewed 2026-08-01T17:57:08.552Z sha 3da8ec5 :: needs maintainer review before merge. :: none
  • reviewed 2026-08-02T13:17:17.964Z sha 3da8ec5 :: needs maintainer review before merge. :: none
  • reviewed 2026-08-02T14:58:07.329Z sha 3da8ec5 :: needs maintainer review before merge. :: none
  • reviewed 2026-08-02T17:06:04.651Z sha 3da8ec5 :: needs maintainer review before merge. :: none

@clawsweeper clawsweeper Bot mentioned this pull request Jul 29, 2026
Open
@blacksmith-sh

This comment has been minimized.

Sign in to view
@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. merge-risk: 🚨 compatibility 🚨 Merging this PR could break existing users, config, migrations, defaults, or upgrades. merge-risk: 🚨 security-boundary 🚨 Merging this PR could weaken sandboxing, authorization, credentials, or sensitive data. and removed rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Jul 29, 2026
@vincentkoc
vincentkoc marked this pull request as ready for review July 29, 2026 20:56
@vincentkoc
vincentkoc requested a review from a team as a code owner July 29, 2026 20:56
@clawsweeper clawsweeper Bot added rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. merge-risk: 🚨 compatibility 🚨 Merging this PR could break existing users, config, migrations, defaults, or upgrades. labels Jul 29, 2026
@clawsweeper clawsweeper Bot mentioned this pull request Jul 31, 2026
Open
@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. and removed rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. labels Jul 31, 2026
@clawsweeper clawsweeper Bot added rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. merge-risk: 🚨 security-boundary 🚨 Merging this PR could weaken sandboxing, authorization, credentials, or sensitive data. and removed status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. merge-risk: 🚨 security-boundary 🚨 Merging this PR could weaken sandboxing, authorization, credentials, or sensitive data. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Jul 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

merge-risk: 🚨 security-boundary 🚨 Merging this PR could weaken sandboxing, authorization, credentials, or sensitive data. P2 Normal priority bug or improvement with limited blast radius. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Support exact OpenClaw candidate packages in native evals

1 participant

@vincentkoc